ACM Interactions

Due to the coronavirus, the biggest conference in our research field (the ACM CHI Conference on Human Factors in Computing Systems) was canceled in April, just a few weeks before it was scheduled to be held in Hawaii. As the Nordics have a strong research community in this area, it was logical to put together an online event for local paper authors. We asked all Nordic authors with accepted CHI 2020 papers if they would like to present; authors of 51 of the 69 papers accepted. To make a manageable event, we needed to set up parallel tracks, giving authors sufficient time to present their work but also enough time for questions and discussion. Online events are much more exhausting to attend than face-to-face meetings, so we made a number of decisions to create an event where participants could practically maintain their attention and energy:

• We ran with 10-minute talks and 10-minute discussion sections. This gave us enough time to deal with technical problems and kept content to a single bite, but also allowed authors to get feedback and engage in discussion. Having 10 minutes for questions turned out to be important because it appears to take an online audience a little longer to digest the talk and formulate questions. Moreover, given the constraints on informal interaction at an online conference like this, we wanted to emphasize and foster interaction among speakers and participants. Having good session chairs was important, as they could start the discussion by asking the first two questions before the audience jumped in. There are other ways of doing this that might be better, but we felt our model kept things closest to the feel of a “real” conference. 


• We limited each session to one hour, with at least 15 minutes break between sessions.


• The event ran “from lunch to lunch” over two days. This maximized participants’ energy levels, and allowed us to run an afterwork event. It also improved the odds that community members from other continents across the world could have a chance to join at least part of the event at a sensible time in their time zones. We added a closing keynote, which also worked well to close the event by bringing everyone together into the same meeting space at the end of the conference. 

What technology to use

If you are going to run a virtual conference, deciding on what applications to use isn’t easy—clearly tools for virtual meetings are evolving quickly and users will have differing preferences. We chose to use Zoom because we had seen it work well before in a few earlier events and because it supports screen sharing and audience participation. While technically the platform worked for us, clearly it is a tool lacking in user-centered design. In particular, the preferences are a minefield, laid with hundreds of poorly worded options across a collection of confusing webpages and dialogue boxes. Moreover, some of these options you really need to get right—or things can go wrong quickly. Feature-wise Zoom is excellent, but if you use it to run an event, you will come to passionately hate its interface—it’s full of idiocies. For example, whenever a presenter shares their screen, by default everyone’s Zoom client also goes to full screen—even if you are just watching the presentation in the background. What other app do you know of that goes full screen without asking you first? But, for better or worse, Zoom is a communication app that most people have installed, and for a public event where you want to encourage participation, choosing another less-familiar system would come at some cost. Zoom also has a “webinar” mode, but we really wanted to have the audience share their video to support participation and interaction as much as possible, so we ran the event as a meeting.

Zoom as a company has also made many deeply dubious decisions, so much so that a lot of people refuse to install the app. So as to provide a second way to access the conference, we also streamed the event to YouTube Live. Again, we chose this because most people are familiar with YouTube. The Zoom integration with YouTube mostly works, except it would occasionally drop the connection and generate a new live feed with a new URL, so we had to frantically change the links on our website to the new URLs. YouTube users could also not ask questions or participate in the meeting beyond spectating. From the audience side of things, though, this Zoom+YouTube combination seemed to work well; the audience could participate and ask questions, and if they had problems with Zoom—or were just shy—they could watch on YouTube while remaining anonymous. Some relied on the combination to enable smoother session-hopping. Needless to say, you need some sort of communication backchannel for real-time communication among organizers as you run an event like this (e.g, Slack). This helps maintain awareness across tracks, troubleshoot quickly if problems arise, and foster a sense of being in it together.

How we set up Zoom and YouTube

We had three parallel tracks, and we had to manage these three tracks while homebound and not actually meeting one another. So, we had three of the organizers each use a dedicated machine to run each of the three tracks. On each machine we created a public Zoom meeting, with no password and no waiting room, that allowed access to those who had registered accounts with Zoom—we wanted to make the event as open as possible. And while this was probably a mistake, it did mean that users didn’t have to register in advance; they could just click the link. As the event was free of charge for participants, we wanted to take the opportunity to welcome everyone—something not possible at our traditional face-to-face conferences. Adding a password wouldn’t have improved security, since we would have had to share the password openly too. As the chair, I also had another machine on the side so that I could move between tracks to monitor how things were going and monitor the Slack channel for organizers. We created new Zoom meetings for each track, keeping them running for the entire day. We didn’t use our own personal meeting room IDs; this meant that each day had a different meeting URL, which caused some problems with participants trying to visit old links. To lock down Zoom, we also made it so participants were muted upon entry to the room. We also restricted screen sharing to just the meeting hosts, and made all the speakers meeting hosts before each session. This was slightly clumsy, because it meant that speakers had to arrive before the session started (which sometimes didn’t happen) and that someone behind the scenes had to watch for when they arrived and quickly make them into co-hosts. After our troubles with zoombombing (see below), we also changed the mute setting so that only the hosts could unmute. We then monitored the event and unmuted people ourselves when they were asking questions. We used the chat feature to let people express that they’d want to ask a question, switching chat settings on and off so as to make sure there wouldn’t be any disturbance during the talks. We forgot to turn off the whiteboard and screen annotation features, which caused us all sorts of problems, so make sure you turn them off.

For each track, the meeting was streamed onto YouTube. YouTube asks for an individual account for each live stream, and also has a waiting period between creating a new account and being able to live stream. To deal with this, in the end we just used our personal YouTube accounts for each live stream. Zoom also seemed to randomly drop the connection to YouTube, which caused the stream to move to a new URL every now and then, leading to the problem of “URL roulette.” 

URL roulette

Before the event, I diligently created redirection URLs (e.g., track1.chinordic2020.org) so that users could just go to that URL and be connected to the relevant Zoom or YouTube feed. Unfortunately, due to the problems with Zoom and YouTube the valid URLs changed quite a bit throughout the day. And with Web redirects you can never be sure if someone will load a cached copy and be sent to an old, invalid URL. We discovered this problem just a few hours before the event started, leading us to frantically change the website and program with the actual links rather than the redirects [1]. The main program for the event was held in a Google doc. This worked great because we could change it and it updated instantly, and as it is a Web app it doesn’t get cached. Unfortunately, in an attempt to be helpful I had also put the URLs onto the website. This was on Google sites, and while it was relatively easy to update, caches meant that we still got a stream of people complaining they were being sent to the wrong URL.

Zoombombing

After the introduction talk, we started the main proceedings in each of our three tracks. It was at this point that our third track was zoombombed. The URL for this track was posted to some sort of forum and we were flooded by kids trying to disrupt the event. They played music from the “Tiger King,” which was kind of funny; showed pornography on their camera—much less funny; and drew swastikas on our first speaker’s presentation—not funny at all. Warning: If this is the content you expect at an academic conference you are going to the wrong sort of academic conferences.

While we probably should have locked down our meeting more beforehand, we also wanted to keep the event open to as many people as possible. We could have set up pre-registration using something like Eventbrite, but sometimes people discover an event on the day it starts and want to drop in. We could have tried to restrict attendance to those from universities, but we also wanted to run an inclusive event. What we did in the end was ask each participant to put in their full name, and kick off those with fake names (the kid who called themself Oswald Mosley at least had some historical knowledge). Any noise or weird videos also got a user kicked off, and we had (thankfully) selected that a user could not rejoin the meeting if they were kicked out. When we were getting attacked, we also enabled the waiting room, and messaged users who were waiting individually to get them to enter a full name before we would let them into the meeting. While this might not be the best way to lock down your meeting (and certainly took a lot of work, with three extra people working behind the scenes when the attack was going on, in addition to the session chair and the organizer hosting the meeting on their machine), it was the best we could come up with quickly. 

Zoom has a particularly stupid feature that allows anyone in a meeting to annotate the screen share — so our zoombombers started drawing onto presenters' slides as they were presenting. There is an option to turn this off somewhere on the huge list of Zoom preferences, but if you don’t turn it off in advance, each presenter needs to go in and turn it off individually. Therefore we had to ask our presenters to do this, before they shared the screen, until we could restart the meeting for the next day. This is one of the many points where it is obvious that Zoom is not really set up for the sort of large event we were running — the annotate feature is so basic it can really only be used for vandalizing others' slides (note to Zoom: a shared notepad feature would actually be useful. Drawing on slides , not so much). Congratulations to Zoom for building a feature that’s really only useful for its Zoombombing users!

In the end, after the amazing teamwork and quick thinking of my co-organizers and others who stepped up to help, the track was delayed by only 20 minutes, and all presenters managed to give their talks and get questions. Congratulations to the amazing team! Zoombombing is pretty horrible, though, particularly for the speakers and chairs who had to deal with this while running the event. Presenting your work is a tough enough experience as it is, so I can only imagine how terrible it is having offensive content thrown at you while trying to do it. It takes place as part of the long depressing history of online violence against women and marginalized minority groups. That those involved did such a good job of reconfiguring the event to deal with it does not take away from the bigger problem.

How to run online talks

Presenting online is challenging, and watching online talks can be, frankly, a little boring. Academic paper presentations are themselves also (surprise!) hit or miss. For this reason, we originally intended to get as many speakers as possible to prerecord their talks, thinking this would get us more polished presentations. Our original plan was to get copies (or links) of videos from our presenters, and then release these to audience members, who could then stream them locally. Thinking this through, though, we couldn’t make sure that everyone would be watching the presentations at the same time, not to mention the challenge of getting everything to work on their own. Attendees might also want to watch the event in the background, so asking them to click on links for each talk could disrupt this. We decided instead to ask presenters to play the videos over Zoom, using the shared screen feature. When we tested this, it worked rather poorly, particularly for videos within presentations. Interestingly, when we actually did this at the conference, the videos worked much more smoothly, suggesting either that something had gone wrong with our testing or that Zoom increases the bandwidth for large events.

Streaming the videos over Zoom, though, makes it impossible to also play the video locally, since there is no way to just turn the audio off on one app—the audio would come from Zoom, and from your video player too. So in the end we just had presenters stream their presentations through Zoom, or present live, and didn’t bother collecting anyone’s videos. This worked well, with some quirks. Presentations are better if you can see the presenter alongside the slides. Prerecording a talk using something like loom.com does a good job of recording your face and displaying it alongside the slides. But if you then play this through Zoom, Zoom also displays a live video of the presenter waiting to answer questions. So you see the presenter twice—once recorded, once live (as they are watching their own talk, something many speakers commented on not enjoying particularly much!). Recorded presentations, although they can be potentially much more professionally done, are also a bit more flat compared to live presentations. There is something energizing about having a presenter actually doing the presentation live, with the audience present, even if they can’t actually technically interact much with the presenter. If I were to run the event again I think I would ask presenters to just present live, as it just gave the event a much better feel. 

Audience participation

We really tried to encourage audience members to stream their video. In Zoom this means that you can get a “gallery” view of those attending. There are lots of reasons for this—it makes the event more sociable, encourages audience members to pay more attention (because you’re being watched yourself), and gives you a chance to see who else is attending what sessions (and a chance to see friends from “across the room”). We kept this optional, since there are lots of good reasons why people cannot or might not want to keep their video on. But the value of having audience members with their cameras on was also a little limited with Zoon’s UI—at maximum you can see only 25 other audience members, and when someone is presenting you can see only five audience members. This takes away from the cache so you can get audience reactions as you are presenting.

We are used to having a round of applause at the end of each talk (and usually after the questions too). We had directed our chairs to be as explicit as possible and ask for the applause at the end of each talk. If the presenter is still presenting their last slide, taking up the screen, most audience members don’t see each other clapping; and because the microphones are muted you sometimes only hear the single applause of the chair—or no clapping at all. While you can see the audience clapping, this creates a slightly discouraging experience. Not having it work well makes you realize that applause is important. Having muted microphones also means that you don’t hear the audience laughing at jokes, heckles, or the like, which again undermines the experience somewhat (living in Glasgow for seven years, I learned what a good heckle is).

Discussions

Being a Nordic event, we wanted to value and encourage participation and discussion. Actually, without discussion what point is there in having an event at all? You could just put a link to pre-recorded videos on a website—clearly the sort of dead, lifeless land occupied by webinars. To avoid that feel, we increased the discussion time for each paper to 10 minutes, shortening the paper presentation time from 15 to 10 minutes. This was also partially because we thought we might lose some time to technical problems and delays (which we actually had very few of in the end), but also to make for a more interesting event compared to just sitting down and reading the papers. 

It is not that paper discussions at conferences are to be held up as the gold standard. Questions are often terrible, and many academics go all “word salad” in their answers. Some great, high-impact papers get no questions when presented because they are just too far ahead of where the audience is thinking. Other papers get acclaim and active discussion because they spark interest among a passionate but confused section of the audience. But questions clearly do give conferences the live feel—there is always the chance of dispute and argument, new ideas emerging through interaction, or authors being called to account for their work.

Doing this online is obviously not going to be as smooth as when everyone is in the same physical room (at least not while we are all still new to online events like this), but it’s crucially important that you get some sort of interaction going—at least a bit of discussion. At our event we made it a rule that to ask a question you had to ask it in the text chat (like standing at the mic), and that the chair would then ask you to unmute and ask your question (or we would unmute the person asking the question and remute them at the end). Originally we had asked for people to type their question, and although I had hoped this might let the chair select the best questions, in the end many people just seemed to type  “I have a question.” We had been warned also that getting questions in an online event can be difficult, so we had prepared the chairs to expect to ask one or two questions themselves. In all cases papers got questions from the audience, but for some it took three or four minutes for the audience to get the questions going. In one or two cases, the chair asked a question and was about to give up, but then four or five questions came in from the audience and a good discussion ensued. Conversation even got heated at times—one questioner got overexcited and muted the session chair so they could ask their question! So while the medium certainly has some limitations, I think we managed to run a participatory event where the audience was actually involved in what was going on. Key lessons here, though, are 1) be patient when waiting for audience questions or interaction, 2) do not worry about silence, and 3) expect session chairs to ask questions, rather than seeing them as the question asker of last resort.

Session chairing

Clearly session chairs were pretty essential to our event. One helpful piece of advice I had been given was the need for session chairs to be more explicit about everything that is going on in the event.  Since not everyone sees everyone else (something the tools don’t help with), participants often have no clue what is going on and where we are in terms of the ceremonies. This means the chair has to be explicit and tell everyone when to clap, when to stop asking questions, when to ask a question, and so on. This can feel like the opposite of a well-run social event where you “don’t see the strings” and can make everything feel rather clumsy, but it is pretty essential for having the event work at all.

In fact, for our event, session chairs might have been better called discussants. The chair had to really know the work enough to throw in a few questions, but also be confident enough to allow the audience to think for a little bit (sometimes as long as a minute), sometimes providing a little filler conversation to give everyone time to catch up. It was also important that alongside the chair there was an organizer running the technical aspects of the meeting and helping coordinate the track across sessions, guiding participants regarding breaks and events in parallel tracks. In the case of our zoombombing, it became clear that it’s better to have two or three such people at the ready to handle and monitor different issues. Running the event is one thing; making the technology work so that the session chair and speakers can focus on their tasks is another. So you need to plan on having at least two people per track to make things run smoothly.

Afterwork

I remember being told at the first conference I went to that “it wasn’t important what talks you went to see but who you didn’t go and see the talks with.” Clearly, we can’t emulate that kind of experience online. Yet, we can at least create some “stuff to talk about” that people can refer to when they eventually do meet up after the event. We scheduled a one-hour afterwork event where we made use of Zoom’s breakout rooms to put people into smaller groups of seven or so people, who chatted for 10 minutes, then went back to the main room and talked a little bit about what had been said in the breakout rooms. To be honest, this was more a structured informal discussion than an “afterwork,” although we did manage to do some singing and dancing at the end (an impromptu performance of Abba, if you must ask). Hardly the full conference experience, but still a chance to catch up a little with others. I am sure there are much better ways of doing this and that we’ll get better at these interactions with more experience.

Why bother?

After reading all this you might wonder if it is worth such an effort in running an online event. Instead of having an online conference you could just put videos up on a website, and even attempt to have some sort of offline discussion around the papers. I am sure that would be a good way of communicating research results, and would be much less effort than having a specific event at a specific time. There are two important elements you would miss with that setup, however. Having an event at a particular time and place gives you the “liveness” of a real event. While you can go back later, there are advantages to actually watching it live, such as taking part in the discussion. Just like most sports fans choose to watch events live, this liveness is something to be valued. Moreover, there are lots of effort/encouragement calculations with having a specific event—it motivates participation (I can just go and watch this thing to get an overview), plus offers the sense of having a shared experience with others. With all the content that is available online, having an event also makes it more likely that people will actually make the time to attend the talks and discussions. Clearly the tools we have right now aren’t really designed to support a good online shared experience, but we hope they will get better over time. 

Acknowledgments

An event like this needs a lot of different people to be involved. In the end we needed five chairs running things behind the scenes: Marianela Ciolfi Felice and Kristina Höök came up with the original idea and shaped the event, then Donald McMillan and Airi Lampinen came on board to help with getting the technology into shape. On the day itself Mareike Glöss, Ville Sundberg, and Asreen Rostami stepped in to deal with the zoombombing. Sara Eriksson, Riyaj Shaikh, and Kasper Karlgren helped with testing the event format and setting up the website. The event relied upon the smooth chairing skills of Mikael Wiberg, Hans Gellersen, Eva Eriksson, Sarah Homewood, Aske Mottelson, Antti Salovaara, Simo Hosio, Juho Pääkkönen, Jessica Cauchard, Marie Louise Juul Sondergaard, Susanne Bodker, Kashyap Todi, Harko Verhagen, Alexandra Weilenmann, and Mikael B. Skov.

Endnotes

1. I later found that my mistake was that I set the redirect up as "permanent 302" not "temporary 301"—if you set it as a temporary redirect this might support rapid changes.

Barry Brown